Why I use RAID1

Why all my desktops have a couple of identical hard disks joined in RAID1

Without any warning sign, I got the message in my command line terminal "New mail for user" last evening. This is not a normal email, but a machine internal message (usually bad news) that I need to read using the terminal command "mail". In this case, each of the 3 partitions from disk sda had failed and were removed from their respective RAID.

I then removed the failed disk and replaced it with another disk of the same size and added it back to the RAID. While doing this, I discovered a small error in the /home partition and corrected it during the night. The system was fully functional in the morning and the RAID was fully synchronized by the end of the day.

All CFD-Lab machines have dual hard disks assembled as RAID1 exactly for this occasion. I set them up as software RAID1 with mdadm in openSUSE, according to the instructions in my CFD-Lab website. In particular, the main user of the machine is listed to receive warnings from mdadm by entering their user name in MDADM_MAIL in openSUSE's YaST /etc/sysconfig Editor. That is how I got the warning without having to constantly check the local mail to root for errors.

Following are all the steps for diagnosing and fixing the disk failure with just a couple of hours of downtime.

1. I confirmed the status of the RAID with
   cat /proc/mdstat
   which indicated the typical failure of one of the RAID components with [_U] .
   I then checked if the failed partitions were still visible with
   lsblk
   Since /dev/sda was completely missing, there was no point in trying to reassemble the RAID right away (sometimes it is just a temporary failure that breaks the RAID and the disk can be quickly reassembled and synchronized).

2. Before continuing, I refreshed my backup to make sure I don't lose any data.
   I updated the backup of my entire home directory to an external disk with my rsync-based script
   syncsendauto
   (see script and details in the Instructions)

3. To remove the failed disk, I checked the serial number of the good disk (sdb) with
   udevadm info --query=all --name=/dev/sdb |grep ID_SERIAL
   Then I shut down the machine, unplugged it, and removed the bad disk.
   I did a final check on the bad disk by inserting it into a USB hard drive dock and all I got was the "click-of-death", confirming its ultimate demise.

4. I replaced the broken disk with another of the same size, but I did not boot up the system right away. I wanted to test the old disk partitions, to make sure they were error free.

5. I booted using a USB with openSUSE and selected "Rescue System". This boots into a minimal Linux system and offers immediately a root prompt.

6. To test the partitions, I first ran a simple check
   fsck /dev/sdbX
   on each partition. Two partitions checked without problems, but partition 2, which contains my home partition, had a problem and automatically started a more complete check that stopped at an inode error and asked if I wanted to fix it. From my experience, when there is one, there might be dozens more similar errors and it becomes very tedious to answer 'y' each time. So, I stopped the test with Ctrl+C. I wanted to restart it with the options 'c' (for read-only scan) and 'y' (for automatic confirmation), but I also wanted to save the output to a file, so I could read through the error messages later, since this process may take hours.

7. Before starting a detailed check of partition 2, I mounted the good partition 1 to /mnt with
   mount /dev/sdb1 /mnt
   so I could save the log file with the errors. Using the 'tee' command I was able to watch the error messages at the same time as they were being recorded in the log file with
   cd /mnt
   fsck -cy /dev/sdb2 | tee disk_errors.txt
   And then I went to sleep, because this check can take many hours.

8. In the morning, the partition was clean (only a few inodes needed fixing) and I could reboot the machine normally.

9. The first step to rebuild the RAID is to format the new disk exactly the same as the other RAID disk.
   In openSUSE, the easiest way is to clone the good RAID disk using YaST Partitioner. In the System View column, select the functioning RAID disk (sdb) and at the bottom of the Overview tab, click on "Partition Table / Clone Partitions to Other Devices", then select the new disk (sda), then OK, Finish.
   Alternatively, you can run as root
   sfdisk -d /dev/sdb | sfdisk /dev/sda

10. Finally, add all the new partitions to their respective RAID with
    mdadm /dev/mdX --add /dev/sdaN
    You can safely execute all add commands, because the system will make sure to only add the next partition once the previous one is completely synchronized.
    You can check the status of the process with
    cat /proc/mdstat
    

You can continue to work normally during synchronization. This is precisely why all my computers have 2 identical hard disks in software RAID1. By the end of the day, the system was fully synchronized and prepared for another disk failure. :)